Edge computing device for controlling electromechanical system with local and remote task distribution control

ABSTRACT

A computing device, including a processor configured to receive sensor data from a control device. The control device may include a control processor configured to execute control instructions to control an actuator of a target electromechanical system and may further include one or more sensors. The processor may identify a first subset of the sensor data and a second subset of the sensor data. The processor may generate first control instructions based on the first subset and transmit the first control instructions to the control processor of the control device. The processor may transmit the second subset to a remote computing device. In response to transmitting the second subset to the remote computing device, the processor may receive a remote processing result from the remote computing device. The processor may generate second control instructions from the remote processing result and transmit the second control instructions to the control processor.

BACKGROUND

Internet of Things (IoT) connected devices have recently come to be usedin a wide variety of settings, such as manufacturing, transportation,resource extraction, climate control for buildings, and biometrics.These IoT connected devices may include sensors with which the IoTconnected devices collect data related to the physical environments inwhich they are located. In addition, IoT connected devices may includecontrollable devices that are configured to perform physical actions inresponse to receiving control signals. The IoT connected device may beconfigured to communicate with a programmable logic controller (PLC),which may be configured to communicate with an edge computing device.The edge computing device may be configured to communicate with aremotely located server computing device, which may provide cloudcomputing resources to the edge computing device.

SUMMARY

According to one aspect of the present disclosure, a computing device isprovided, including a processor configured to receive sensor data from acontrol device. The control device may include a control processorconfigured to execute control instructions to control an actuator of atarget electromechanical system. The control device may further includeone or more sensors configured to collect the sensor data from aphysical environment in which the target electromechanical system issituated. The processor may be further configured to identify a firstsubset of the sensor data to be processed locally and a second subset ofthe sensor data to be processed remotely. The processor may be furtherconfigured to generate first control instructions based on the firstsubset of the sensor data. The processor may be further configured totransmit the first control instructions to the control processor of thecontrol device. The processor may be further configured to transmit thesecond subset of the sensor data to a remote computing device. Inresponse to transmitting the second subset of the sensor data to theremote computing device, the processor may be further configured toreceive a remote processing result from the remote computing device. Theprocessor may be further configured to generate second controlinstructions from the remote processing result and transmit the secondcontrol instructions to the control processor of the control device.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. Furthermore,the claimed subject matter is not limited to implementations that solveany or all disadvantages noted in any part of this disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A schematically shows a computing system including a computingdevice connected to a remote computing device such as a server via aremote connection over a computer network such as a wide area network,and also connected to one or more sensors and a target electromechanicalsystem via a local connection such as a serial, fieldbus, or LANconnection, according to one example embodiment.

FIG. 1B shows the computing device of FIG. 1A in an example in which thecomputing device is configured to control a plurality of roboticdevices.

FIG. 2 shows the task distribution module executed by the processor ofthe computing device in additional detail, according to the embodimentof FIG. 1A.

FIG. 3 shows an example parameter space diagram for sensor inputs,according to the embodiment of FIG. 1A.

FIG. 4A schematically shows the computing device when a predictiveprocessing model is executed, according to the embodiment of FIG. 1A.

FIG. 4B shows sensor data and predicted sensor data when the predictiveprocessing model of FIG. 4A is executed.

FIG. 5 schematically shows the computing device and the remote computingdevice when the remote processing result received at the computingdevice includes a second instruction generation function, according tothe embodiment of FIG. 1A.

FIG. 6A shows a flowchart of a method for use with a computing device tocontrol one or more actuators of a target electromechanical system,according to the embodiment of FIG. 1A.

FIGS. 6B-6C show additional steps of the method of FIG. 6A that may beperformed when the first subset of the sensor data to be processedlocally and the second subset of the sensor data to be processedremotely are identified.

FIG. 6D shows additional steps of the method of FIG. 6A that may beperformed when the second control instructions are generated.

FIG. 7 shows a schematic view of an example computing environment inwhich the computing device of FIG. 1A may be enacted.

DETAILED DESCRIPTION

In the conventional IoT system discussed above, in which an IoTcontrolled device is controlled using a PLC that communicates with anedge computing device, communication latency between the PLC and theedge computing device may interfere with the operation of IoT controlleddevice. This latency may result in a low temporal resolution for signalstransmitted to and from the IoT controlled device and may limit whattypes of control schemes may be used. Limitations on the bandwidth ofcommunications between the PLC and the edge computing device may furtherlimit the resolution of the data received from sensors and the controlschemes that may be used in the control signals sent to controllabledevices.

In order to address the above shortcomings of conventional IoT systems,a computing system 1 including a computing device 10 is provided, asschematically shown in the example of FIG. 1A. The computing device 10of FIG. 1A may include a processor 12. In addition, the computing device10 may include memory 14, which may include volatile memory andnon-volatile memory. The computing device 10 may be configured totransmit control instructions to a target electromechanical system 30.The computing device 10 may be located proximate to the targetelectromechanical system 30 in the physical environment and maycommunicate with the target electromechanical system 30 via a wired orwireless local connection. The wired or wireless local connection istypically a serial connection (e.g., universal serial bus (USB),RS-232), fieldbus connection (e.g., GPIB, MAP, MMS, MODBUS, PROFIBUS,INTERBUS, CAN, etc.), local area network (LAN) connection (e.g., wiredLAN over Ethernet/IP or wireless LAN), or other suitable localconnection that does not include a high latency computer network such asa wide area network (WAN). The target electromechanical system mayinclude, to name a few examples, a robot, HVAC system, alarm system,plumbing system, conveyor system, manufacturing applicator, or othercontrolled system. These examples are not to be considered limiting asnumerous other examples are possible. The computing device 10 may bereferred to as an edge computing device due to its position on thenetwork topology having a local connection to the targetelectromechanical system 30 and a remote connection to a remotecomputing device 40, described below.

The processor 12 of the computing device 10 may be configured to executecontrol instructions to control one or more actuators 32 included in thetarget electromechanical system 30. The one or more actuators 32 may beone or more components of the target electromechanical system 30 thatare configured to move when they receive electrical signals. Forexample, the one or more actuators 32 may include one or more motors,one or more speakers, one or more electrically controllable switches, orone or more other types of actuators. One or more controllablecomponents other than actuators, such as one or more heating coils,lights, cameras, conveyors, valves, etc., may be included in the targetelectromechanical system 30 in some examples.

The processor 12 may be further configured to receive sensor data 50from one or more sensors 26 configured to collect the sensor data 50from a physical environment 20 in which the target electromechanicalsystem 30 is situated. Although the one or more sensors 26 are shown inFIG. 1A as being provided separately from the target electromechanicalsystem 30, the one or more sensors 26 may be integrated into the targetelectromechanical system 30 in other examples. The one or more sensors26 may, for example, include one or more cameras, one or moretemperature sensors, one or more accelerometers, one or more pressuresensors, one or more microphones, one or more magnetic field sensors, orone or more other types of sensors. The processor 12 may be configuredto receive the sensor data 50 via a local connection selected from thegroup consisting of a serial connection, a fieldbus, and a local areanetwork.

Subsequently to receiving the sensor data 50 from the one or moresensors 26, the processor 12 may be further configured to identify afirst subset 52 of the sensor data 50 to be processed locally and asecond subset 54 of the sensor data 50 to be processed remotely. Asdiscussed in further detail below, the second subset 54 may betransmitted to a remote computing device 40. In some examples, the firstsubset 52 and the second subset 54 may overlap, whereas in otherexamples, the first subset 52 and the second subset 54 may be disjoint.

FIG. 1B shows the computing device 10 in an example in which thecomputing device 10 is used with two target electromechanical systems 30in the form of two robotic arms. In the example of FIG. 1B, each of thetarget electromechanical systems 30 includes two actuators 32 configuredto receive control instructions from the computing device 10. Theactuators 32 in the example of FIG. 1B are each motor configured to movejoints of the robotic arms. In addition, each of the robotic armsincludes two sensors 26 that are configured to transmit sensor data 50to the computing device 10. For example, the sensors 26 included in eachof the robotic arms may be accelerometers. The computing device 10 inthe example of FIG. 1B is configured to communicate with the remotecomputing device 40 over a network 46.

The processor 12 of the computing device 10 may be configured to executea task distribution module 56 configured to perform local and remotetask distribution control, by identifying tasks or data associated withtasks that are to be processed locally at computing device 10 orremotely at remote computing device 40. Thus, the task distributioncontrol model 56 is configured to identify the first subset 52(processed locally) and the second subset 54 (processed remotely) of thesensor data 50. The task distribution module 56 may include a firsttrained machine learning model 70. Additionally or alternatively, thetask distribution module 56 may include one or more task distributionrules 71 that may be applied to the sensor data 50 to sort the sensordata 50 into the first subset 52 and the second subset 54.

FIG. 2 shows the task distribution model 56 in additional detail. Asshown in FIG. 2, when the processor 12 executes the task distributionmodel 56, the processor 12 may be further configured to identify a firsttask set 63 including one or more processing tasks 61 that areconfigured to be performed on the first subset 52 of the sensor data 50and a second task set 65 including one or more processing tasks 61 thatare configured to be performed on the second subset 54 of the sensordata 50. When the processor 12 transmits the second subset 54 of thesensor data 50 to the remote processing device 40, the processor 12 maybe further configured to transmit an indication of the second processingset 65 to the remote processing device 40. Alternatively, the one ormore processing tasks 61 included in the second processing set 65 may beidentified at the remote computing device 40.

The processor 12 may be configured to sort the sensor data 50 into thefirst subset 52 and the second subset 54 based at least in part on oneor more system conditions 55, one or more task properties 57, or one ormore performance criteria 58. The one or more system conditions 55 areproperties of the computing device 10 that may affect computingperformance. For example, the one or more system conditions 55 mayinclude a local processor load 55A of the processor 12, a memorycapacity 55B of the memory 14, or a network latency 55C of a networkconnection between the computing device 10 and the remote computingdevice 40. The one or more task properties 57 may be properties of theone or more processing tasks 61 included in the first task set 63 or thesecond task set 65. For example, the one or more task properties 57 mayinclude one or more of an estimated processing cost 57A, an estimatedmemory usage 57B, and an estimated network usage 57C. The one or moretask properties 57 may be indicated for one or more processing tasks 61included in the first task set 63 or the second task set 65. The one ormore performance criteria 58 may, for example, include one or morerespective task priority levels 58A for the one or more processing tasks61. The one or more performance criteria 58 may additionally oralternatively include one or more latency criteria 58B, which mayindicate one or more maximum allowed network latencies for the one ormore processing tasks 61. Additionally or alternatively to the systemconditions 55, task properties 57, and performance criteria 58 providedin the above examples, other conditions or settings may be used asinputs by the task distribution module 56 when determining the firstsubset 52 and the second subset 54 of the sensor data 50.

In some examples, as discussed above, the processor 12 may be configuredto identify the first subset 52 of the sensor data 50 and the secondsubset 54 of the sensor data 50 at least in part at a first trainedmachine learning model 70. For example, the first trained machinelearning model 70 may be a recurrent neural network. The first trainedmachine learning model 70 may be configured to receive one or morevalues of the sensor data 50 as input, and may be configured to outputone or more respective classifications for the one or more input valuesthat classify the one or more input values as being included in thefirst subset 52 or the second subset 54.

In some examples, as shown in the example parameter space diagram 80 ofFIG. 3, the processor 12 may be configured to identify the first subset52 of the sensor data 50 at least in part by determining that the sensordata 50 included in the first subset 52 is within a first input-spaceregion 82. The first input-space region 82 is a region of a parameterspace defined by the variables of the sensor data 50 received from theone or more sensors 26. The parameter space may have a respectivedimension for each dimension of the sensor data 50. In the example ofFIG. 3, the variables of the sensor data 50 are a first motor speed anda second motor speed, which may, for example, be measured via imagingwith one or more cameras. The processor 12 may be further configured toidentify the second subset 54 of the sensor data 50 at least in part bydetermining that the sensor data 50 included in the second subset 54 isoutside the first input-space region 82. In the example parameter spacediagram of FIG. 3, the second subset 54 of the sensor data 50 is insteadlocated within a second input-space region 84. The first input-spaceregion 82 may, for example, be a region of parameter space values thatindicate that the target electromechanical system 30 is operating withina normal operating mode. In such examples, when the processor 12identifies sensor data 50 located outside the first input-space region82, that sensor data 50 may indicate that the target electromechanicalsystem 30 is operating under atypical conditions. Although the parameterspace shown in the parameter space diagram 80 of FIG. 3 is a space withtwo variables, the parameter space may alternatively have some othernumber of variables.

In some examples in which the first subset 52 and the second subset 54are identified at a first trained machine learning model 70, the firsttrained machine learning model 70 may be a predictive processing model,as shown in the example of FIGS. 4A-4B. FIG. 4A schematically shows thecomputing device 10 when a predictive processing model is executed. Asdepicted in FIG. 4A, the processor 12 may be configured to generatepredicted sensor data 90 at the first trained machine learning model 70.The sensor data 50 and the predicted sensor data 90 are shown inadditional detail in FIG. 4B. As shown in FIG. 4B, the sensor data 50may include respective values 53 for each of one or more variables 51.Each of those values 53 may be associated with a respective time value55 at which that value 53 was measured. The predicted sensor data 90 mayinclude one or more predicted values 93 of the sensor data 50. The oneor more predicted values 93 may be associated with one or more timevalues 55 at which the one or more respective values 53 of the one ormore variables 51 included in the sensor data 50 are predicted to matchthose predicted values 93. When sensor data 50 is received for those oneor more time values 55, the processor 12 may be further configured todetermine a prediction error level 92 for the predicted sensor data 90relative to the sensor data 50 received from the one or more sensors 26at that time value 55. For example, the prediction error level 92 may bea mean squared error for the one or more predicted values 93 of thepredicted sensor data 90. In some examples, the processor 12 may beconfigured to determine the prediction error level 92 for a timeinterval including a plurality of time values 55.

Returning to FIG. 4A, the processor 12 may be further configured toidentify, as the first subset 52, sensor data 50 below a predeterminedprediction error threshold 94, and identify, as the second subset 54,sensor data 50 above the predetermined prediction error threshold 94.Thus, sensor data 50 that is accurately predicted by the first trainedmachine learning model 70 may be processed locally, while sensor data 50that exhibits high levels of prediction error may be transmitted to theremote computing device 40 for processing.

In some examples, when the sensor data 50 and the predicted sensor data90 each have a dimensionality of two (as one particular example) orgreater, the processor 12 may be configured to compute respectiveprediction error levels 92 for one or more subsets of the plurality ofvariables 51. For example, the processor 12 may be configured to computerespective prediction error levels 92 for each of the measured variables51 individually, or for one or more subsets of the plurality ofvariables 51 that include two or more variables 51. The processor 12 maybe further configured to compare the prediction error levels 92 for eachof those subsets of the plurality of variables 51 to a respectivepredetermined prediction error threshold 94. For each subset of theplurality of variables that exceeds its corresponding predeterminedprediction error threshold 94, the one or more variables in that subsetmay be transmitted to the remote computing device 40. The othervariables 51 of the plurality of variables 51 may be processed locally.

In examples in which the first subset 52 of the sensor data 50 and thesecond subset 54 of the sensor data 50 are selected at a first trainedmachine learning model 70, the processor 12 may be further configured tofurther train the first trained machine learning model 70 based on thesensor data 50. In examples in which the processor 12 is configured togenerate predicted sensor data 90 at the first trained machine learningmodel 70, the processor 12 may be further configured to modify the firsttrained machine learning model 70 when sensor data 50 is received. Sucha modification may be made based on the prediction error level 92 forthat sensor data 50. For example, when the first trained machinelearning model is configured to generate predicted sensor data 90, thefirst trained machine learning model may be initially trained viaunsupervised learning using a set of training sensor data and may befurther updated via unsupervised learning using the sensor data 50.

Returning to FIG. 1A, the processor 12 may be further configured togenerate first control instructions 60 based on the first subset 52 ofthe sensor data 50. Subsequently to generating the first controlinstructions 60, the processor 12 may be further configured to executethe first control instructions 60 to control the one or more actuators32 of the target electromechanical system 30. The first controlinstructions 60 may be instructions to perform one or more controloperations at the one or more actuators 32 included in the targetelectromechanical system 30. The first control instructions 60 may, forexample, be an instruction to start, stop, or change the direction of amotor; to open or close an electromechanical switch; to play a sound ona speaker; or perform some other operation at the one or more actuators32 of the target electromechanical system 30. These examples are to beconsidered illustrative and not limiting as numerous functions arepossible.

The first control instructions 60 may be generated at an instructiongeneration module 59, which may include a second trained machinelearning model 72. Additionally or alternatively, the instructiongeneration module 59 may include one or more instruction generationrules 73. As shown in the example of FIG. 5, the processor 12 may beconfigured to apply a first instruction generation function 66 to thefirst subset of the sensor data 50 to compute the first controlinstructions 60. In examples in which the processor 12 is configured togenerate the first control instructions 60 at least in part at a secondtrained machine learning model 72, the second trained machine learningmodel 72 may, for example, be a recurrent neural network.

The processor 12 may be further configured to transmit the second subset54 of the sensor data 50 to a remote computing device 40. For example,the processor 12 may be configured to transmit the second subset 54 ofthe sensor data 50 to the remote computing device 40 via a wide areanetwork (WAN). The remote computing device 40 may include a remoteprocessor 42 and remote memory 44. The functionality of the remotecomputing device 40 may be distributed between a plurality of remotecomputing devices 40 configured to communicate with each other. In someexamples, the remote computing device 40 may be a server computingdevice located in a data center.

In response to transmitting the second subset 54 of the sensor data 50to the remote computing device 40, the processor 12 may be furtherconfigured to receive a remote processing result 64 from the remotecomputing device 40. The processor 12 may be further configured togenerate second control instructions 62 from the remote processingresult 64 and execute the second control instructions 62 to control theone or more actuators 32 of the target electromechanical system 30.Alternatively, the processor 12 may be configured to receive the secondcontrol instructions 62 from the remote computing device 40 as theremote processing result 64.

In some examples, the remote processing result 64 may include additionalsensor data received at the remote computing device 40 via one or moreother computing devices. The one or more other computing devices may beconnected to one or more additional sensors located in the physicalenvironment 20. In such examples, the second control instructions 62 maybe generated based on the additional sensor data. Thus, the additionalsensor data may be used as contextual sensor data that provides theprocessor 12 with additional information about the physical environment20 in which is it located. The additional sensor data may, for example,indicate that a temperature measured by an additional sensor connectedto another computing device is close to the top of a safe operatingtemperature range. In response to receiving a remote processing result64 that includes the temperature measurement, the processor 12 may beconfigured to generate and execute second control instructions 62 toturn on a fan.

In some examples, the remote processing result 64 may include anindication of the second input-space region 84 within which the sensordata 50 included in the second subset 54 is located. The remoteprocessing result 64 may further include, as shown in FIG. 5, a secondinstruction generation function 68 that maps one or more sensor datavalues 53 located within the second input-space region 84 to one or morerespective control instructions. In such examples, the processor 12 maybe further configured to generate the second control instructions 62 atleast in part by applying the second instruction generation function 68to the second subset 54 of the sensor data 50. The processor 12 may befurther configured to store the second instruction generation function68 in the memory 14. Thus, as the processor 12 receives sensor data 50located within new regions of the parameter space, the area of theparameter space for which control instructions are generated withoutoffloading processing to the remote computing device 40 may expand.

Additionally or alternatively to including a second instructiongeneration function 68, the remote processing result 64 may include amodification to the first instruction generation function 66 with whichthe processor is configured to generate the first control instructions60. For example, when the first instruction generation function is asecond trained machine learning model 72, the modification to the firstinstruction generation function 66 may be additional training performedat the second trained machine learning model 72.

The remote computing device 40 may be configured to generate the remoteprocessing result 64 at a remote instruction generation module 75, asshown in the example of FIG. 1A. The remote instruction generationmodule 75 may include a remote machine learning model 76. Additionallyor alternatively, the remote instruction generation module 75 mayinclude one or more remote instruction generation rules 76. The sensordata 50 included in the second subset 54 may be used as input for theremote machine learning model 74 or the one or more remote instructiongeneration rules 76 included in the remote instruction generation module75. The remote machine learning model 74 or the one or more remoteinstruction generation rules 76 may additionally take as input theadditional sensor data received from the one or more other computingdevices. In examples in which the first trained machine learning model70 is a predictive processing model, as shown in the example of FIG. 4A,the remote machine learning model 74 may be further configured toreceive the prediction error level 92 for the predicted sensor data 90.In other examples, the remote machine learning model 74 may be a versionof the second trained machine learning model 72 with additional trainingand/or additional model parameters.

In some examples, the processor 12 may be configured to generate thefirst control instructions 60 based on the first subset 52 of the sensordata 50 in real time as the first subset 52 is received. The processor12 may be further configured to transmit the second subset 54 of thesensor data 50 to the remote computing device 40 for non-real-timeprocessing. Thus, real-time control of the one or more actuators 32 maybe performed at the computing device 10 with low latency, whereas otherprocessing tasks may be performed at the remote computing device 40 withhigher latency but increased processing power. It will be appreciatedthat the connection between the computing device 10 and the remotecomputing device 40 is typically over a WAN.

In some examples, as shown in the example of FIG. 5, the processor 12may be configured to identify the first subset 52 of the sensor data 50and the second subset 54 of the sensor data 50 based at least in part ona communication latency 96 between the processor 12 and the remotecomputing device 40. For example, the processor 12 may be configured tocompute a latency threshold 98 based on the sensor data 50 in real timeas the sensor data is received. The latency threshold 98 may bedetermined at the first trained machine learning model 70. When thelatency threshold 98 is below the communication latency between theprocessor 12 and the remote computing device 40, the processor 12 may befurther configured to compute first control instructions 60 locallybased on the sensor data 50. When the latency threshold 98 is above thecommunication latency 96, the processor 12 may be further configured totransmit at least a portion of the sensor data 50 to the remotecomputing device 40 for processing. Thus, the processor 12 may beconfigured to determine control instructions locally when the sensordata 50 indicates urgent conditions and to transmit sensor data 50 tothe remote computing device 40 for processing when the sensor data 50does not indicate urgent conditions. The first subset 52 and the secondsubset 54 may be identified based at least in part on the communicationlatency 96 in examples in which the remote machine learning model 74 isa version of the second trained machine learning model 72 with moremodel parameters.

FIG. 6A shows a flowchart of a method 100 for use with a computingdevice. The method 100 of FIG. 6A may be performed at the computingdevice 10 of FIG. 1A or at some other computing device. At step 102, themethod 100 may include receiving sensor data from one or more sensorsconfigured to collect the sensor data from a physical environment inwhich a target electromechanical system is situated. For example, thesensor data may be received from one or more cameras, one or moretemperature sensors, one or more accelerometers, one or more pressuresensors, one or more microphones, one or more magnetic field sensors, orone or more other types of sensors. The computing device may be locatedin the physical environment in which the one or more sensors areprovided and may receive the sensor data via a wired or wirelessconnection such as a LAN.

At step 104, the method 100 may further include identifying a firstsubset of the sensor data to be processed locally and a second subset ofthe sensor data to be processed remotely. The first subset and thesecond subset may be identified based on the values of the sensor data.Additionally, or alternatively, the first subset and the second subsetmay be identified based at least in part on one or more systemconditions of the computing device, one or more task properties ofprocessing tasks associated with the first subset and the second subset,or one or more performance criteria for the computing device or a remotecomputing device at which the second subset of the sensor data isconfigured to be processed.

FIG. 6B shows additional steps of the method 100 that may be performedwhen performing step 104. At step 118, step 104 may include determiningthat the sensor data included in the first subset is within a firstinput-space region. In addition, at step 120, step 104 may furtherinclude determining that the sensor data included in the second subsetis outside the first input-space region. The first input-space regionmay be a region of a parameter space formed by the one or more variablesmeasured by the one or more sensors.

In some examples, at step 122, step 104 may include identifying thefirst subset of the sensor data and the second subset of the sensor databased at least in part on a communication latency between the computingdevice and the remote computing device. In such examples, thecommunication latency may be compared to a latency threshold that isdynamically computed in real time at the computing device and indicateshow time-sensitive the sensor data is. The first subset may be a subsetof the sensor data for which the latency threshold is below thecommunication latency (indicating that the sensor data has high timesensitivity), and the second subset may be a subset of the sensor datafor which the latency threshold is above the communication latency(indicating that the sensor data has low time sensitivity).

FIG. 6C also shows additional steps of the method 100 that may beperformed when step 104 is performed. At step 124, the method 100 mayinclude identifying the first subset of the sensor data and the secondsubset of the sensor data at least in part at a first trained machinelearning model. The first trained machine learning model may, forexample, be a recurrent neural network. The first trained machinelearning model may take the sensor data as an input and may outputrespective classifications for the sensor data values as being includedin the first subset or the second subset.

In some examples, at step 126, step 124 may include generating predictedsensor data at the first trained machine learning model. The predictedsensor data may be time series data. In addition, at step 128, step 124may further include determining a prediction error level for thepredicted sensor data relative to the sensor data received from the oneor more sensors. Separate prediction error levels may, in some examples,be determined for individual variables of the sensor data or for one ormore subsets of a plurality of variables included in the sensor data. Inother examples, one prediction error level may be determined for thesensor data at a corresponding time value or time interval. At step 130,step 124 may further include identifying, as the first subset, sensordata for which the prediction error level is below a predeterminedprediction error threshold. In addition, at step 132, step 124 mayfurther include identifying, as the second subset, sensor data for whichthe prediction error level is above the predetermined prediction errorthreshold.

In some examples in which step 124 is performed, the method 100 mayfurther include, at step 134, further training the first trained machinelearning model based on the sensor data. For example, when the firstmachine learning model is a predictive processing model configured togenerate predicted sensor data, the first machine learning model may befurther trained via online learning as it receives sensor data, and maythereby increase the accuracy of the predicted sensor data it generates.

Returning to FIG. 6A, the method 100 may further include, at step 106,generating first control instructions based on the first subset of thesensor data. In some examples, the first control instructions may begenerated at least in part at a second trained machine learning model.At step 108, the method 100 may further include executing the firstcontrol instructions to control one or more actuators of the targetelectromechanical system. Thus, instructions based on the first subsetof the sensor data may be used to control the one or more actuatorswithout incurring delays that would result from communicating with theremote computing device.

At step 110, the method 100 may further include transmitting the secondsubset of the sensor data to a remote computing device. The secondsubset may be transmitted to the remote computing device via a WAN. Insome examples, the method 100 may further include, at step 112A,receiving a remote processing result from the remote computing device inresponse to transmitting the second subset of the sensor data to theremote computing device. In examples in which step 112A is performed,the method 100 may further include, at step 114, generating secondcontrol instructions from the remote processing result. In otherexamples, the method 100 may alternatively include, at step 112B,receiving the second control instructions from the remote computingdevice in response to transmitting the second subset of the sensor datato the remote computing device.

At step 116, the method 100 may further include executing the secondcontrol instructions to control the one or more actuators of the targetelectromechanical system. Thus, the remote processing result may be usedto select actions performed at the one or more actuators.

FIG. 6D shows additional steps of the method 100 that may be performedin some examples when the second control instructions are generated atstep 114. At step 136, generating the second control instructions atstep 114 may further include, based on the remote processing result,modifying a first instruction generation function with which the firstcontrol instructions are generated. In examples in which step 136 isperformed, the method 100 may further include, at step 138, applying themodified first instruction generation function to the second subset ofthe sensor data.

In some examples, the remote processing result may include a secondinstruction generation function. In such examples, step 114 may furtherinclude, at step 140, applying the second instruction generationfunction included in the remote processing result to the second subsetof the sensor data.

In one example use case scenario, the computing device may be used in asewage treatment plant. In such an example, the processor of thecomputing device may be configured to receive sensor data indicating aflow rate for effluent through a line with flow rate controlled by avalve. The computing device may be configured to transmit controlinstructions to an actuator that controls the valve, thereby functioningas part of a flow control system. The computing device may beprogrammable with a set point for the flow rate through the line, andmay be configured to transmit control signals to adjust the valveopening to control the effluent to the flow rate. When the flow ratedeviates from the set point by an amount greater than a correspondingthreshold, the computing device may transmit sensor data to a remotecomputing device for further processing. For example, the sensor datatransmitted to the remote computing device may include the flow rate andvalve position during a time interval starting at a predetermined amountof time before the time value at which the threshold was exceeded. Theremote computing device may input the sensor data it receives into aremote machine learning model, which may generate control instructionsthat are transmitted back to the computing device. The computing devicemay execute those control instructions to control the actuators includedin the valves, and may thereby restore the flow rate to normal levels.

In another example, the computing device of the example above may beconfigured to execute a task distribution rule that executes locally alocal flow rate control task for a particular flow rate in a particularline as controlled by a particular valve as the target electromechanicalsystem, and that sends the flow rate and valve position information tothe remote machine learning model locally to execute a global flowcontrol task, such providing drainage to a city sewer system while notoverloading a downstream outlet channel. To execute the global controltask, the remote machine learning model receives as inputs the flowrates and valve positions of other lines within the system, as well asthe level of the outlet channel. In this manner, local control tasks areperformed locally at the computing device, and remote control tasks areperformed under the direction of the remote computing device, whichtakes into account a wider variety of information sources.

In another example, the computing device may be configured to receiveimages from a camera imaging a manufacturing line as a sensor locally,generate a downscaled lower resolution of the image, and process thelower resolution version of the image using a convolutional neuralnetwork (CNN) to output a first control instruction to a robotperforming a weld on a target workpiece in the image. The CNN may betrained to perform rough defect detection identify a defect in theworkpiece based on the lower resolution image and stop the weldingoperation by the robot in real time during the operation. The computingdevice may be configured to send a higher resolution version of theimage to the remote computing device for processing, and a convolutionalneural network at the remote computing device may be configured todetect fine defects in the higher resolution version of the image, andsend an instruction to computing device to take a remedial action. Theremedial action may be to control the robot to perform a second weld torepair the defect, or if the workpiece has moved on in the assemblyprocess, to instruct a downstream manufacturing conveyor to route theworkpiece back for repair or discard the workpiece due to the finedefect.

Using the devices and methods discussed above, one or more actuatorsincluded in an IoT connected device may be controlled while avoidingcommunication delays that would otherwise occur when a PLC communicateswith an edge computing device. Since such communication delays may beavoided, the computing device may be provided with additional computingcapabilities, such as implementation of machine learning models, thatwould not be available on a PLC. The computing device may also offloadprocessing of some of the sensor data it receives to a remote computingdevice. For example, the remote computing device and the computingdevice may have a “brain-spinal cord” relationship in which computingtasks with low computing power requirements and high time sensitivityare performed at the computing device whereas tasks with high computingpower requirements and low time sensitivity are performed at the remotecomputing device. Thus, the computing device may quickly and efficientlycontrol the IoT controlled device while also being able to respondappropriately to a wide variety of sensor inputs.

In some embodiments, the methods and processes described herein may betied to a computing system of one or more computing devices. Inparticular, such methods and processes may be implemented as acomputer-application program or service, an application-programminginterface (API), a library, and/or other computer-program product.

FIG. 7 schematically shows a non-limiting example of a computing system200 that can enact one or more of the devices and methods describedabove. Computing system 200 is shown in simplified form. Computingsystem 200 may embody the computing device 10 described above andillustrated in FIG. 1. Computing system 200 may take the form of one ormore personal computers, server computers, tablet computers,home-entertainment computers, network computing devices, gaming devices,mobile computing devices, mobile communication devices (e.g., smartphone), and/or other computing devices, and wearable computing devicessuch as smart wristwatches and head mounted augmented reality devices.

Computing system 200 includes a logic processor 202 volatile memory 204,and a non-volatile storage device 206. Computing system 200 mayoptionally include a display subsystem 208, input subsystem 210,communication subsystem 212, and/or other components not shown in FIG.7.

Logic processor 202 includes one or more physical devices configured toexecute instructions. For example, the logic processor may be configuredto execute instructions that are part of one or more applications,programs, routines, libraries, objects, components, data structures, orother logical constructs. Such instructions may be implemented toperform a task, implement a data type, transform the state of one ormore components, achieve a technical effect, or otherwise arrive at adesired result.

The logic processor may include one or more physical processors(hardware) configured to execute software instructions. Additionally oralternatively, the logic processor may include one or more hardwarelogic circuits or firmware devices configured to executehardware-implemented logic or firmware instructions. Processors of thelogic processor 202 may be single-core or multi-core, and theinstructions executed thereon may be configured for sequential,parallel, and/or distributed processing. Individual components of thelogic processor optionally may be distributed among two or more separatedevices, which may be remotely located and/or configured for coordinatedprocessing. Aspects of the logic processor may be virtualized andexecuted by remotely accessible, networked computing devices configuredin a cloud-computing configuration. In such a case, these virtualizedaspects are run on different physical logic processors of variousdifferent machines, it will be understood.

Non-volatile storage device 206 includes one or more physical devicesconfigured to hold instructions executable by the logic processors toimplement the methods and processes described herein. When such methodsand processes are implemented, the state of non-volatile storage device206 may be transformed—e.g., to hold different data.

Non-volatile storage device 206 may include physical devices that areremovable and/or built-in. Non-volatile storage device 206 may includeoptical memory (e.g., CD, DVD, HD-DVD, Blu-Ray Disc, etc.),semiconductor memory (e.g., ROM, EPROM, EEPROM, FLASH memory, etc.),and/or magnetic memory (e.g., hard-disk drive, floppy-disk drive, tapedrive, MRAM, etc.), or other mass storage device technology.Non-volatile storage device 206 may include nonvolatile, dynamic,static, read/write, read-only, sequential-access, location-addressable,file-addressable, and/or content-addressable devices. It will beappreciated that non-volatile storage device 206 is configured to holdinstructions even when power is cut to the non-volatile storage device206.

Volatile memory 204 may include physical devices that include randomaccess memory. Volatile memory 204 is typically utilized by logicprocessor 202 to temporarily store information during processing ofsoftware instructions. It will be appreciated that volatile memory 204typically does not continue to store instructions when power is cut tothe volatile memory 204.

Aspects of logic processor 202, volatile memory 204, and non-volatilestorage device 206 may be integrated together into one or morehardware-logic components. Such hardware-logic components may includefield-programmable gate arrays (FPGAs), program- andapplication-specific integrated circuits (PASIC/ASICs), program- andapplication-specific standard products (PSSP/ASSPs), system-on-a-chip(SOC), and complex programmable logic devices (CPLDs), for example.

The terms “module,” “program,” and “engine” may be used to describe anaspect of computing system 200 typically implemented in software by aprocessor to perform a particular function using portions of volatilememory, which function involves transformative processing that speciallyconfigures the processor to perform the function. Thus, a module,program, or engine may be instantiated via logic processor 202 executinginstructions held by non-volatile storage device 206, using portions ofvolatile memory 204. It will be understood that different modules,programs, and/or engines may be instantiated from the same application,service, code block, object, library, routine, API, function, etc.Likewise, the same module, program, and/or engine may be instantiated bydifferent applications, services, code blocks, objects, routines, APIs,functions, etc. The terms “module,” “program,” and “engine” mayencompass individual or groups of executable files, data files,libraries, drivers, scripts, database records, etc.

When included, display subsystem 208 may be used to present a visualrepresentation of data held by non-volatile storage device 206. Thevisual representation may take the form of a graphical user interface(GUI). As the herein described methods and processes change the dataheld by the non-volatile storage device, and thus transform the state ofthe non-volatile storage device, the state of display subsystem 208 maylikewise be transformed to visually represent changes in the underlyingdata. Display subsystem 208 may include one or more display devicesutilizing virtually any type of technology. Such display devices may becombined with logic processor 202, volatile memory 204, and/ornon-volatile storage device 206 in a shared enclosure, or such displaydevices may be peripheral display devices.

When included, input subsystem 210 may comprise or interface with one ormore user-input devices such as a keyboard, mouse, touch screen, or gamecontroller. In some examples, the input subsystem may comprise orinterface with selected natural user input (NUI) componentry. Suchcomponentry may be integrated or peripheral, and the transduction and/orprocessing of input actions may be handled on- or off-board. Example NUIcomponentry may include a microphone for speech and/or voicerecognition; an infrared, color, stereoscopic, and/or depth camera formachine vision and/or gesture recognition; a head tracker, eye tracker,accelerometer, and/or gyroscope for motion detection and/or intentrecognition; as well as electric-field sensing componentry for assessingbrain activity; and/or any other suitable sensor.

When included, communication subsystem 212 may be configured tocommunicatively couple various computing devices described herein witheach other, and with other devices. Communication subsystem 212 mayinclude wired and/or wireless communication devices compatible with oneor more different communication protocols. As non-limiting examples, thecommunication subsystem may be configured for communication via awireless telephone network, or a wired or wireless local- or wide-areanetwork. In some examples, the communication subsystem may allowcomputing system 200 to send and/or receive messages to and/or fromother devices via a network such as the Internet.

The following paragraphs discuss several aspects of the presentdisclosure. According to one aspect of the present disclosure, acomputing device is provided, including a processor configured toreceive sensor data from one or more sensors configured to collect thesensor data from a physical environment in which a targetelectromechanical system is situated. The processor may be furtherconfigured to identify a first subset of the sensor data to be processedlocally and a second subset of the sensor data to be processed remotely.The processor may be further configured to generate first controlinstructions based on the first subset of the sensor data. The processormay be further configured to execute the first control instructions tocontrol one or more actuators of the target electromechanical system.The processor may be further configured to transmit the second subset ofthe sensor data to a remote computing device, and, in response totransmitting the second subset of the sensor data to the remotecomputing device, receive a remote processing result from the remotecomputing device. The processor may be further configured to generatesecond control instructions from the remote processing result. Theprocessor may be further configured to execute the second controlinstructions to control the one or more actuators of the targetelectromechanical system.

According to this aspect, the processor may be configured to identifythe first subset of the sensor data at least in part by determining thatthe sensor data included in the first subset is within a firstinput-space region. The processor may be further configured to identifythe second subset of the sensor data at least in part by determiningthat the sensor data included in the second subset is outside the firstinput-space region.

According to this aspect, the remote processing result may include anindication of a second input-space region within which the sensor dataincluded in the second subset is located. The remote processing resultmay further include a second instruction generation function that mapsone or more sensor data values located within the second input-spaceregion to one or more respective control instructions. The processor maybe further configured to generate the second control instructions atleast in part by applying the second instruction generation function tothe second subset of the sensor data.

According to this aspect, the remote processing result may include amodification to a first instruction generation function with which theprocessor is configured to generate the first control instructions.

According to this aspect, the processor may be configured to identifythe first subset of the sensor data and the second subset of the sensordata at least in part at a first trained machine learning model.

According to this aspect, the processor may be further configured to, atthe first trained machine learning model, generate predicted sensordata. The processor may be further configured to determine a predictionerror level for the predicted sensor data relative to the sensor datareceived from the one or more sensors. The processor may be furtherconfigured to identify, as the first subset, sensor data for which theprediction error level is below a predetermined prediction errorthreshold. The processor may be further configured to identify, as thesecond subset, sensor data for which the prediction error level is abovethe predetermined prediction error threshold.

According to this aspect, the processor may be further configured tofurther train the first trained machine learning model based on thesensor data.

According to this aspect, the processor may be configured to generatethe first control instructions at least in part at a second trainedmachine learning model.

According to this aspect, the processor may be configured to generatethe first control instructions based on the first subset of the sensordata in real time as the first subset is received. The processor may befurther configured to transmit the second subset of the sensor data tothe remote computing device for non-real-time processing.

According to this aspect, the processor may be configured to identifythe first subset of the sensor data and the second subset of the sensordata based at least in part on a communication latency between theprocessor and the remote computing device.

According to this aspect, the processor may be configured to receive thesensor data via a local connection selected from the group consisting ofa serial connection, a fieldbus, and a local area network. The processormay be configured to transmit the second subset of the sensor data tothe remote computing device via a wide area network (WAN).

According to another aspect of the present disclosure, a method for usewith a computing device is provided. The method may include receivingsensor data from one or more sensors configured to collect the sensordata from a physical environment in which a target electromechanicalsystem is situated. The method may further include identifying a firstsubset of the sensor data to be processed locally and a second subset ofthe sensor data to be processed remotely. The method may further includegenerating first control instructions based on the first subset of thesensor data. The method may further include executing the first controlinstructions to control one or more actuators of the targetelectromechanical system. The method may further include transmittingthe second subset of the sensor data to a remote computing device, and,in response to transmitting the second subset of the sensor data to theremote computing device, receiving a remote processing result from theremote computing device. The method may further include generatingsecond control instructions from the remote processing result. Themethod may further include executing the second control instructions tocontrol the one or more actuators of the target electromechanicalsystem.

According to this aspect, identifying the first subset of the sensordata may include determining that the sensor data included in the firstsubset is within a first input-space region. Identifying the secondsubset of the sensor data may include determining that the sensor dataincluded in the second subset is outside the first input-space region.

According to this aspect, generating the second control instructions mayinclude, based on the remote processing result, modifying a firstinstruction generation function with which the first controlinstructions are generated. Generating the second control instructionsmay further include, applying the modified first instruction generationfunction to the second subset of the sensor data.

According to this aspect, the method may further include identifying thefirst subset of the sensor data and the second subset of the sensor dataat least in part at a first trained machine learning model.

According to this aspect, the method may further include, at the firsttrained machine learning model, generating predicted sensor data. Themethod may further include determining a prediction error level for thepredicted sensor data relative to the sensor data received from the oneor more sensors. The method may further include identifying, as thefirst subset, sensor data for which the prediction error level is belowa predetermined prediction error threshold. The method may furtherinclude identifying, as the second subset, sensor data for which theprediction error level is above the predetermined prediction errorthreshold.

According to this aspect, the first control instructions may begenerated at least in part at a second trained machine learning model.

According to this aspect, the first control instructions may begenerated in real time based on the first subset of the sensor data asthe first subset of sensor data is received. The second subset of thesensor data may be transmitted to the remote computing device fornon-real-time processing.

According to this aspect, the method may further include identifying thefirst subset of the sensor data and the second subset of the sensor databased at least in part on a communication latency between the computingdevice and the remote computing device.

According to another aspect of the present disclosure, a computingdevice is provided, including a processor configured to receive sensordata from one or more sensors configured to collect the sensor data froma physical environment in which a target electromechanical system issituated. The processor may be further configured to identify a firstsubset of the sensor data to be processed locally and a second subset ofthe sensor data to be processed remotely. The processor may be furtherconfigured to generate first control instructions based on the firstsubset of the sensor data. The processor may be further configured toexecute the first control instructions to control one or more actuatorsof the target electromechanical system. The processor may be furtherconfigured to transmit the second subset of the sensor data to a remotecomputing device, and, in response to transmitting the second subset ofthe sensor data to the remote computing device, receive second controlinstructions from the remote computing device. The processor may befurther configured to execute the second control instructions to controlthe one or more actuators of the target electromechanical system.

It will be understood that the configurations and/or approachesdescribed herein are exemplary in nature, and that these specificembodiments or examples are not to be considered in a limiting sense,because numerous variations are possible. The specific routines ormethods described herein may represent one or more of any number ofprocessing strategies. As such, various acts illustrated and/ordescribed may be performed in the sequence illustrated and/or described,in other sequences, in parallel, or omitted. Likewise, the order of theabove-described processes may be changed.

The subject matter of the present disclosure includes all novel andnon-obvious combinations and sub-combinations of the various processes,systems and configurations, and other features, functions, acts, and/orproperties disclosed herein, as well as any and all equivalents thereof.

1. A computing device comprising: a processor configured to: receivesensor data from one or more sensors configured to collect the sensordata from a physical environment in which a target electromechanicalsystem is situated; identify a first subset of the sensor data to beprocessed locally and a second subset of the sensor data to be processedremotely; generate first control instructions based on the first subsetof the sensor data; execute the first control instructions to controlone or more actuators of the target electromechanical system; transmitthe second subset of the sensor data to a remote computing device; inresponse to transmitting the second subset of the sensor data to theremote computing device, receive a remote processing result from theremote computing device; generate second control instructions from theremote processing result; and execute the second control instructions tocontrol the one or more actuators of the target electromechanicalsystem.
 2. The computing device of claim 1, wherein the processor isconfigured to: identify the first subset of the sensor data at least inpart by determining that the sensor data included in the first subset iswithin a first input-space region; and identify the second subset of thesensor data at least in part by determining that the sensor dataincluded in the second subset is outside the first input-space region.3. The computing device of claim 2, wherein: the remote processingresult includes: an indication of a second input-space region withinwhich the sensor data included in the second subset is located; and asecond instruction generation function that maps one or more sensor datavalues located within the second input-space region to one or morerespective control instructions; and the processor is further configuredto generate the second control instructions at least in part by applyingthe second instruction generation function to the second subset of thesensor data.
 4. The computing device of claim 1, wherein the remoteprocessing result includes a modification to a first instructiongeneration function with which the processor is configured to generatethe first control instructions.
 5. The computing device of claim 1,wherein the processor is configured to identify the first subset of thesensor data and the second subset of the sensor data at least in part ata first trained machine learning model.
 6. The computing device of claim5, wherein the processor is further configured to: at the first trainedmachine learning model, generate predicted sensor data; determine aprediction error level for the predicted sensor data relative to thesensor data received from the one or more sensors; identify, as thefirst subset, sensor data for which the prediction error level is belowa predetermined prediction error threshold; and identify, as the secondsubset, sensor data for which the prediction error level is above thepredetermined prediction error threshold.
 7. The computing device ofclaim 5, wherein the processor is further configured to further trainthe first trained machine learning model based on the sensor data. 8.The computing device of claim 1, wherein the processor is configured togenerate the first control instructions at least in part at a secondtrained machine learning model.
 9. The computing device of claim 1,wherein the processor is configured to: generate the first controlinstructions based on the first subset of the sensor data in real timeas the first subset is received; and transmit the second subset of thesensor data to the remote computing device for non-real-time processing.10. The computing device of claim 1, wherein the processor is configuredto identify the first subset of the sensor data and the second subset ofthe sensor data based at least in part on a communication latencybetween the processor and the remote computing device.
 11. The computingdevice of claim 1, wherein: the processor is configured to receive thesensor data via a local connection selected from the group consisting ofa serial connection, a fieldbus, and a local area network; and theprocessor is configured to transmit the second subset of the sensor datato the remote computing device via a wide area network (WAN).
 12. Amethod for use with a computing device, the method comprising: receivingsensor data from one or more sensors configured to collect the sensordata from a physical environment in which a target electromechanicalsystem is situated; identifying a first subset of the sensor data to beprocessed locally and a second subset of the sensor data to be processedremotely; generating first control instructions based on the firstsubset of the sensor data; executing the first control instructions tocontrol one or more actuators of the target electromechanical system;transmitting the second subset of the sensor data to a remote computingdevice; in response to transmitting the second subset of the sensor datato the remote computing device, receiving a remote processing resultfrom the remote computing device; generating second control instructionsfrom the remote processing result; and executing the second controlinstructions to control the one or more actuators of the targetelectromechanical system.
 13. The method of claim 12, wherein:identifying the first subset of the sensor data includes determiningthat the sensor data included in the first subset is within a firstinput-space region; and identifying the second subset of the sensor dataincludes determining that the sensor data included in the second subsetis outside the first input-space region.
 14. The method of claim 12,wherein generating the second control instructions includes: based onthe remote processing result, modifying a first instruction generationfunction with which the first control instructions are generated; andapplying the modified first instruction generation function to thesecond subset of the sensor data.
 15. The method of claim 12, furthercomprising identifying the first subset of the sensor data and thesecond subset of the sensor data at least in part at a first trainedmachine learning model.
 16. The method of claim 15, further comprising:at the first trained machine learning model, generating predicted sensordata; determining a prediction error level for the predicted sensor datarelative to the sensor data received from the one or more sensors;identifying, as the first subset, sensor data for which the predictionerror level is below a predetermined prediction error threshold; andidentifying, as the second subset, sensor data for which the predictionerror level is above the predetermined prediction error threshold. 17.The method of claim 12, wherein the first control instructions aregenerated at least in part at a second trained machine learning model.18. The method of claim 12, wherein: the first control instructions aregenerated in real time based on the first subset of the sensor data asthe first subset of sensor data is received; and the second subset ofthe sensor data are transmitted to the remote computing device fornon-real-time processing.
 19. The method of claim 12, further comprisingidentifying the first subset of the sensor data and the second subset ofthe sensor data based at least in part on a communication latencybetween the computing device and the remote computing device.
 20. Acomputing device comprising: a processor configured to: receive sensordata from one or more sensors configured to collect the sensor data froma physical environment in which a target electromechanical system issituated; identify a first subset of the sensor data to be processedlocally and a second subset of the sensor data to be processed remotely;generate first control instructions based on the first subset of thesensor data; execute the first control instructions to control one ormore actuators of the target electromechanical system; transmit thesecond subset of the sensor data to a remote computing device; inresponse to transmitting the second subset of the sensor data to theremote computing device, receive second control instructions from theremote computing device; and execute the second control instructions tocontrol the one or more actuators of the target electromechanicalsystem.